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ABSTRACT 

This research study was conducted to examine particular 
metacognitive monitoring and regulatory skills in the context of solving 
science problems in a computer-based learning environment. An effort was also 
made to extend the work of H. Swanson (1990) using a different measure for 
metacognition and multiple measures of ability, achievement, and aptitude. 
Participants were 1,502 students in grades 5 through 9 from schools across 
the United States. Knowledge of Cognition, Objectivity, and Problem 
Representation predicted successful problem solving, as hypothesized. In 
terms of confirming and extending Swanson *s work, this study found congruent 
results: high levels of metacognitive self-regulation compensated for low 
overall abilities. The research also confirmed Swanson's assertion that the 
constructs of metacognition and ability, achievement, and aptitude operate as 
independent processes. Results further demonstrate that metacognitive self- 
regulation was a better predictor of success at problem solving than many 
standardized measures used in classrooms across the country. (Contains 1 
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Research states that metacognitive monitoring and control are important skills for successful problem solving (e.g., Artzt 
& Armour-Tliomas 1 992: Carr & Jessup, 1 997; Hmelo & Cole, 1 996; Tobias & Everson, 1 995). In particular, research has 
demonstrated that certain .self- regulatory and metacognitive skills predict problem solving success as well as or better than 
traditional predictors of general ability such as achievement scores (Swanson, 1 990). 

The purpose of this research study was twofold. First, we sought to examine particular metacognitive monitoring and 
regulatory skills in the context of solving science problems in a computer-based learning environment. Second we sought to 
confirm and extend Swan.son's (1 990) work using a different measure for metacognition and multiple measures of ability, 
achievement, and aptitude. 

There were several noteworthy findings. Knowledge of Cognition, Objectivity, and Problem Representation predicted 
successful problem solving, as hypothesized. In terms of confirming and extending Swanson’s (1 990) work, we found congruent 
result.s — high levels of metacognitive self-regulation compensated for low overall abilities. This research also confirmed 
Swanson’s a.s.sertion that the constructs of metacognition and ability, achievement, and aptitude operate as independent proces.ses. 
Results further demonstrated that metacognitive self-regulation was a better predictor of success at problem solving than many 
.standardized measures used in classrooms across the country. 

It is our hope that such research will eventually lead to the development of a model for problem-solving activities and 
computer-ba.sed materials that would foster metacognitive self-regulation. 



Iniroduciion 



With the current emphasis in education on information management and the advent of the Internat as an educational 
re.soLirce, there is a great need for students to be able to self-regulate how they learn and acquire new skills. Research conducted 
over the last 1 5 years on self-regulated learning has primarily focused on three core components: metacognitive awarene.ss, 
.strategy u.se, and motivational control (Bruning, Schraw & Ronning, 1 995). In the present study we focused on the influence of 
metacognitive awareness for effective use of problem solving strategies 

There were two primary research goals. First, we examined five sub-components of metacognitive self-regulation as 
de.scribed by Howard, McGee, Shia, & Hong (2000) and their influence upon problem solving. The sub-components are: 

Knowledge of Cognition Measures how much learners understand about their unique cognitive abilities and the ways 
they learn best. Includes an awareness of one's own learning and memory processes as well as learning strengths and 
weaknesses. 

Objectivity Defined as standing outside oneself and thinking about one’s learning as it proceeds. Includes an awareness 
of one's learning goals and alternative choices in accomplishing a learning goal. 

Problem Representation Includes an awareness of strategies one uses to understand the problem fully before proceeding. 
Subtask Monitorimz Defined as breaking the problem down into subtasks and monitoring the completion of each subtask. 
Evaluation Measures the degree to which students are aware ol‘ checking their work throughout the entire problem- 
.solving process to evaluate if it is being done correctly. 

Second, we sought to extend prior research in the area of metacognitive self-regulation and how it interacts with student 
ability, achievement, and aptitude in the context of solving problems, in 1 990 H. Lee Swanson presented a pivotal work linking 
metacognition to successful problem solving. Swanson set out to demonstrate the independence of metacognition and general 
aptitude on various problem-solving measures. He defined aptitude as the propensity to successfully perform school-related tasks, 
and measured it using standardized, cognitive ability and achievement tests. He measured metacognitive ability using tape- 
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recorded responses to a metacog ni live questionnaire. His findings indicated that metacognition was more important for problem- 
solving success than aptitude and that in situations where students had low aptitudes but high metacognition, they performed as 
well as students who had high aptitude. Howard, McGee, Hong and Shia (2000) found results that coincided with Swanson's 
(1990) findings in the context of students learning through computer-ba.sed inquiry. Their research showed that those students with 
high nietacognitive .self-regulation compensated for low achievement on content understanding and problem solving measures. 
They used GPA as a measure of achievement. The ramifications of both of these studies are important given the strong emphasis 
educators have placed on ability indicators like aptitude and achievement by educators throughout the history of psychological 
measurement. 

Hypotheses 

1) All five sub-components of nietacognitive self-regulation as measured by the Inventory of Metacognitive Self-Regulation 
(Howard, McGee, & Shia 1 999) will predict success at problem solving. 

2) High levels of nietacognitive .self-regulation will compensate for low overall achievement, ability, or aptitude. 

In Swanson's (1990) work, he used two standardized measures to determine aptitude; the Cognitive Abilities Test (Thorndike 
& Hagen, 1978) and the Comprehensive Test of Basic Skills (1978). In Howard, McGee, Hong and Shia (2000), they used GPA as 
a measure of achievement. In the present study we sought to extend this research by using multiple measures of achievement, 
ability, and aptitude. Our subject pool was comprised of approximately fifteen hundred students in classrooms acro.ss the United 
States. In addition to collecting data on nietacognitive self-regulation and problem solving outcomes, we asked teachers to also 
send any available data from student achievement, ability and aptitude scores already collected. A .summary of the measures is 
presented in Table 1 . 

Table 1 

De.scriptions of Re.search Instruments and Number of Participants for Each 



rest 


Pull Nniiie iiiul Version 


N 


De.scription 


IM.SR 


Inventory of Metacognitive Self- 


LS.16 


Measures five metacognitive self-regiiialory variables using a Likert-scale format: Knowledge 




Regulation 




of Cognition. Objectivity, Problem Representation, Subtask Monitoring and Evaluation. For 
ages 12-18. 


■ 


Problem Solving measure 


1 502 


Assesses problem solving skills such as di awing conclusions and making inferences. For grades 
5-12. 








Achievement 


- 


Content Understanding 


1420 


Assessment of achievement within tlie pi esent instructional context— normalized across the 
subject population. 


CTB.S 


Comprehensive Test of Basic Skills. 


29 


Norm-referenced standardized achievement test. Subscales include: Language Arts, 




Level C 




Mathematics, Science and Social Studies. 


CTPIM 


Comprehensive Testing Program III. 


126 


Norm-referenced test for grades 6-8. Subscales include: Vocabulary. Reading Comprehension. 




Level B 




Writing Mechanics, Writing Process, and Mathematics. 


CPA 


Grade Point Average 


1 184 


Student grade point avei age, normalized across the subject population. 


SAT 9 


Stanford Achievement Test- 9th Bdition. 


228 


Criterion-referenced standardized achievement test. Subscales include: Reading, Math, 




S Bonn, Intermediate Level 




Language, and Science. 








Science Achievement 


ITBS 


Iowa Test of Basic Skills- Science 


.57 


Norm-referenced constructed-response test which asses.ses strategic thinking and problem 




Performance Assessment, Grades 8 & 9 




solving capabilities in the domain of science. 


MPAP 


Michigan Educational A.ssessment 


164 


Criterion-referenced test which measures what students know and what students are able to do. 




Program- Science Subscale 




Given in grades 5 and 8. 


BOT 


Seventh Grade Science Essential 


124 


Criterion-referenced test given in 7th grade. Developed by the Rochester, NY School District to 




Outcomes Test 




iiieasure science achievement. 








Cognitive Ability 


Cornell 


Cornell Critical Thinking Measure, .Ird 


44 


Measures specific critical thinking abilities: Induction, Credibility, Deduction, and 




Bdition, 
Level X 




Identification of Assumptions. For grades 5 through college-level. 


MA'r 


Matrix Analogies Test, Short Form 


40 


Measures nonverbal reasoning ability. For ages 5-17. 


TBR 


Test of Everyday Reasoning 


46 


Measures critical thinking judgements made in everyday situations. For grades 6 and up. 








Aptitude 


OLSAT 


Otis-Lennon School Ability Test, 7th 


48 


Measures cognitive reasoning skills which relate to a student’s ability to learn and succeed in 




Bdition, Level B 




school. Level B is for students in grades 4 and 5. 
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Method 

Panicipanis 

Participants included 1502 students, grades 5-9, Irom schools across the United States. They represented a cross-section 
of socioeconomic backgrounds and urban/suburban/mral categorizations. The ethnic breakdown included 36.5% Caucasian, 4.7% 
Asian American, 2.3% African American, 1.3% Hispanic or Latino, and 55.2% other or not reported. By gender, the breakdown 
was 45.7% female, 47.6% male and 6.7% not reported. 

Procedure and Materials 

Learning Outcomes 

Students used Astronomy Village® Investigating the Solar SystenT^ soltware for approximately 20 instructional days 
(see McGee & Howard, 1999 for a description of Astronomy Villaiie^) . Students were assessed on two learning outcomes: Content 
Understanding , and Problem Solving . The Content Understanding measure focused on the core requirements lor life and planetary 
processes as presented in the software. In the present study Content Understanding was used as an independent variable 
representing achievement within this particular learning context. The Problem Solving measure focused on drawing conclusions 
from data and making inferences about planetary processes. The instrument was validated as part of the Astronomy Village 
development project sponsored by the National Science Foundation (McGee, Howard, Dimitrov, Hong, & Shia, 2001) 

Inventory of Metacognitive Self-Regulation 

At pretest time, students took the Inventory of Metacognitive Self-Regulation (IMSR) which measures five factors related 
to awareness of learning processes and control of learning strategies: (1) Knowledge of Cognition, (2) Objectivity (3) Problem 
Representation, (4) Subtask Monitoring, and (5) Evaluation (Howard, McGee & Shia, 1999). The IMSR includes 32 items that use 
a five-point Likert scale. For each of the 32 items, students are instructed to circle the answer that best described the way they 
solve problems in math or science class (l=never, 2=seldom/rarely, 3=sometimes, 4=often/freqiiently, 5=always). The validation 
of the IMSR is discussed elsewhere (Howard, McGee, Shia, & Hong, 2000). 

Achievement. Coiinitive Ability, and Aptitude Measures 

We collected achievement, cognitive ability, and aptitude scores from classroom subsets of the total 1502 students 
according to Table 1. Achievement indicators included both norm-referenced and criterion-referenced standardized tests including 
the Comprehensive Test of Basic Skills, Comprehensive Testing Program IIJ, and Stanford Achievement Test. We placed student 
grade point average and the Content Understanding measure in the achievement category as well. Three of the tests in this 
category measured achievement specifically in the area of science (Iowa Test of Basic Skills, Michigan Educational Assessment 
Program, and the Seventh Grade Essential Outcome Test). Cognitive ability measures included the Cornell Critical Thinking 
Measure, the Matrix Analogies Test, and the Test of Everyday Reasoning. The Otis-Lennon School Ability Test measures 
aptitude and cognitive reasoning skills which relate to a student’s ability to learn and succeed in school. Achievement and aptitude 
measures are presented in national percentile scores. A description of each inventory is provided in Table I as well. For ease of 
discussion, we will collectively call these instruments "ability" measures. 

Results 

For all analyses, we u.sed the general linear model (GLM), which is a unifying analysis technique that includes both 
analysis of variance and multiple linear regression. When the analysis involved examining differences, ANOVA results are 
reported. When the analysis involved examining relationships, multiple regression results are reported. 

Initial analyses revealed that both gender and age were significant predictors of a student's overall metacognitive self- 
regulation score (b = -. 1 37, p < ().()()() 1 , and b = .06 1 , p < 0.000 1 , respectively). In this case, females had significantly higher Total 
IMSR scores. In all subsequent analyses, variability due to age and gender was removed. 

Our first hypothesis was that each of the five sub-components of metacognitive self-regulation would predict success at 
problem solving. Overall, the IMSR total score was a significant predictor of Problem Solving, B = . 149, p < 0.0001. See Table 2 
for more details. In addition, three of the five factors (Knowledge of Cognition, Objectivity, & Problem Representation) were also 
significant predictors (B = .082, p =.0217, B = -.07 1, p =.0402, and B = . 1 15, p =.0021 , respectively). 

Table 2 

Summary of Multiple Regression Analysis: IMSR Total and Subscales as Predictors of Success at Problem Solving tN=1417) 



Variable CoclTicicm SE Traiio a value 



IMSR 


.25 


.04 


.S..S.S 


< .000 1 * 


Knowledge of Cognition 


.6,1 


.28 


2..10 


.022* 


Objeetivity 


-.46 


.22 


-2.0.'i 


.040* 


Problem Recognition 


.77 


.25 


.1.09 


.002* 


Subta.sk Monitoring 


..10 


.21 


I.IO 


.270 


Evaluation 


.08 


.2.1 


0..1.S 


.724 



Noie . IMSR = InveiUory of Melacogniiive Self- Regulation 
*u<.o.s. 
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Our second hypothesis was that high levels ol' metacognitive sell-regulation would compensate for low overall 
achievement, ability, or aptitude. Swanson (1990) conducted 2X2 ANOVAs, categorizing students according to metacognitive 
ability and aptitude (High v. Low Metacognition X High v. Low Aptitude). His results revealed an interaction between the High 
Metacognition/Low Aptitude group and the Low Metacognition/High Aptitude group. In order to test for similar interactions, we 
also used 2X2 ANOVAs — one I'or each ol'the twelve instruments. According to our hypothesis, one would expect a significant 
interaction between these two groups, with the High Metacognition/Low Ability group outperforming the Low 
Metacognition/High Ability group. 

In order to group students into High and Low Metacognitive Sell-Regulation and High and Low Ability, we used mean 
scores as the cutoff, in the case of student GPA, we had a large enough subject pool that we used the top and bottom 27‘’’ 
percentiles. 

Mean .scores on Problem Solving for each cell in the 2X2 are presented in Table 3 for each instrument. The interaction 
statistics between the two cells of interest are also shown. Using a= .004 to account for the Type I error rate due to numerous 
analyses, six of twelve instruments showed significant interactions between the cells of interest. These six included Content 
Understanding, the CTP III, the SAT 9, the EOT, the TER and the OLSAT. These six span all three categories of achievement, 
cognitive ability, and aptitude. It should be noted that with the exception olThe MEAP, the High Melacognition/Low Ability group 
had higher mean scores than the Low Metacognilion/High Ability group in every instance. This is illustrated in Figure 1. 

Table 3 

Summary of Mean Scores on Problem Solving as a Function of High and Low MetacoLmition and High and Low Ability, with 
Selected Interaction Statistics 



Test 


HM/HA 


HM/LA 


LM/HA 


LM/LA 


Scheffe Post Hoc Tests for Interaction 
(HM/LA Compared to LM/HA) 

SE ]2 value 








Achievement 






Content 


1 1.08 


10.74 


7.38 


6.56 


0.36 


<0.0001* 


CTBS 


8.34 


10.68 


7.91 


7.89 


1.16 


0.021 


CTP III 


19.89 


19.98 


16.57 


17.67 


0.85 


0.0001* 


GPA 


10.49 


9.75 


8.68 


6.71 


0.83 


0.195 


SAT9 


10.56 


10.62 


7.08 


7.97 


0.57 


<0.0001* 








Science Achievement 






ITBS 


12.81 


12.05 


10.53 


1 1.58 


2.01 


0.455 


MEAP 


7.58 


6.36 


8.03 


6.80 


0.80 


0.039 


EOT 


9.03 


10.00 


6.47 


6.35 


0.78 


<0.0001* 








Cognitive 


Ability 






Cornell 


14.54 


15.65 


14.28 


15.12 


2.38 


0.568 


MAT 


13.44 


12.17 


8.89 


7.90 


1.58 


0.045 


TER 


19.07 


19.33 


14.04 


13.00 


1.65 


0.003* 








Aptitude 






OLSAT 


16.30 


18.00 


12.15 


12.70 


1.89 


0.003* 



Note . HM = High Metacognition; LM = Low Metacognition; HA = High Ability; LA = Low Ability. 
< .004. 
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Figure 1. Mean score on problem solving as a function grouping and instrument 
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Discussion 

The Five Suh-Componenls of Melacognilive Sell- Regulation 

The lad that ihree of the five metacognitive self-regulation SLib-components (Knowledge of Cognition, Objectivity, & 
Problem Representation) were important for problem solving has considerable implications for theory related to self- regulated 
learning. First, the results indicate that Knowledge of Cognition is important for problem solving. To our knowledge, this is the 
first empirical study to report this connection. It is unclear in the research whether the acquisition and use of problem solving 
strategies necessarily depends on one's knowledge about or awareness of their cognition. Although many assume the mindful use 
of regulatory processes may precede effective use of problem solving strategies, this may not be the case. For example, consider 
the situation where problem solving strategies have become automatized (Sternberg, 1986). Awareness of one's learning processes 
may not be necessary for effective learning (Brown, 1987; Hacker, 1998; Kanfer & Ackerman, 1989). 

Second, the research indicates that Objectivity and Problem Representation are important predictors for successful 
problem solving. Objectivity — the notion of standing outside oneself and thinking about one’s learning — is often discussed as an 
important goal of interventions which teach learning strategies. Is one's objectivity an innate ability or is it a skill to be taught? 
Clearly more research into this sub-component of metacognitive self-regulation is warranted. Many researchers have described and 
studied the process of problem representation (e.g., Bransford & Stein, 1984; Ellis & Siegler, 1994; Hayes, 1978; Flunt, 1994; 
Reimann & Chi, 1989; Sternberg, 1986). In fact, Jonassen (2000) stated that problem representation is the most critical element for 
problem solving. The Problem Representation sub-component, as defined by Howard and colleagues (2000), measures the degree 
to which one is aware of strategies they use to understand a given problem. If becoming aware of the need to understand a problem 
before proceeding is important for successful problem solving, perhaps future research should investigate methods by which this 
awareness can be taught or at least supported through the instructional materials. 

The fact that Subtask Monitoring and Evaluation were not predictors of problem solving is understandable in retrospect. 
Monitoring and evaluation tasks may not have been necessary for success at problem solving because the problems presented were 
of a short-term nature. On the assessment, the learner only needed to read a scenario and apply problem solving skills in one 
instance to solve the problem and then move on. Monitoring and evaluation would not be needed under these conditions to do 
well. 

So what do the results mean for the design of instruction? There are two directions that emerge. First, if Knowledge of 
Cognition, Objectivity, and Problem Representation are important precursors to successful problem solving, instruction should be 
tailored to foster these skills. Since some believe that metacognitive abilities are innate and therefore slow to change, how to do 
this would be a research question of capital importance. White and Fredriksen (1998) and Davis (1996; 1998; 1999) made in-roads 
in this area which demonstrated how to promote the development of metacognitive skills through well-placed prompts and 
journaling activities. One implication from such research is that if it is possible to train students in utilizing metacognitive 
strategies, this training can help students to succeed in spite of low ability levels, achievement, or aptitude. Second, it may behoove 
instructional designers to consider how to develop problem solving activities to compensate for students' metacognitive 
weaknesses. For example, a student who is unaware of their cognitive strengths and limitations as they pertain to the task at hand, 
could be prompted to complete a worksheet, or have a discussion with their peers about choosing learning strategies before 
proceeding. 
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MelacoLMiilive SeU-Re^^ulalion v. Ability 

Swanson’s research has been heralded as a crucial turning point for research in metacognition and problem solving. The 
research reported here provides an important confirmation of Swanson’s findings and extends it into two new areas. In terms of 
confirming Swanson’s (1990) work, we found congruent results; high levels of metacognitive self-regulation compensated for low 
overall abilities. In addition, our research confirmed Swanson's assertion that the constructs of metacognition and ability, 
achievement, and aptitude operate as independent processes. Furthermore, the results lend additional construct validity to the self- 
report measure of metaeognitive self-regulation, the IMSR (Howard, McGee, & Shia, 1999). 

In terms of extending Swanson’s (1990) work, we found that Knowledge of Cognition, Objectivity, and Problem 
Representation were important self- regulatory variables for effeetive problem solving. We also found that the results withstood the 
scrutiny of using various measures of ability, achievement, and aptitude. Six of twelve measures showed a significant interaction 
between High Metacognition/ Low Ability and Low Metacognition/ High Ability groups. The results demonstrated that 
metacognitive self-regulation was a better predictor of success at problem solving than these standardized measures used in 
classrooms across the country. 

As shown by the results, students with high abilities may not need to utilize metacognitive self-regulation as much as 
those of lesser ability. This idea is consistent with prior research which found metacognition to be unimportant for familiar 
learning environments (Schwartz, Andersen, Howard, Hong & McGee, 1998). 
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